I told him I wanted to end it and he refused.
Doesn't this mean he really only cares about him, not really about your feelings?
If you want to end it, you can. That's your decision. You don't need his permission. He has absolutely no say in that. Just as you can't force him to stay with you if he doesn't want to.
Does he disregard your input and opinion in other things as well?
From the rest of what you write it's clear he doesn't really want you to be part of his life long term. Believe him when he says it isn't gonna work out long term.
The man is telling you he won't be with you for long. He just want a sexual friendship with no commitment. He told you. Believe what he told you.
Now if you are fine being friends with benefits for as long as it lasts, by all means have fun.
If you're not fine with that, leave. It's your decision, and your decision only.
Don't stick around to see if you can change his mind, you can't. Believe what the man told you.
The above is all just about human interaction. In this case, it gets even worse because there's a cult involved. If you are anything less than absolutely sure that both you and him want to be committed to each other for a long time, leave.
Yes, that hurts. But it's better than wasting your precious time with someone who doesn't want to be committed to you if that is what you desire.
Good luck!